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Abstract:  In  this  paper,  an  analytical  expression  is  developed  for  the  differential  entropy  of 
a  sinusoid  with  a  Beta-distributed  phase  angle.  This  particular  signal  model  is  prevalent  in 
optical  communications,  however  an  expression  for  the  associated  differential  entropy  does 
not  currently  exist.  The  expression  we  derive  is  approximate  as  it  relies  on  a  series  expansion 
for  one  of  the  key  terms  needed  in  the  derivation.  However,  we  are  able  to  show  that  the 
approximation  is  accurate  (error  <  5%)  for  a  wide  variety  of  Beta  parameter  choices. 
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1.  Introduction 

The  concept  of  entropy  as  a  measure  of  information  was  introduced  by  Shannon  [1]  and  has  since 
been  discussed  in  many  books  (e.g.,  see  [2],  [3]).  Entropy  has  played  a  central  role  in  the  field  of 
communications,  providing  limits  on  both  data  compression  and  channel  capacity  [2].  Entropy-based 
signal  processing  techniques  and  analysis  have  also  enjoyed  a  great  deal  of  success  in  a  diverse  set 
of  applications  ranging  from  ecological  system  monitoring  [4]  to  crystallography  [5].  Successful 
application  of  entropy-based  approaches  is  often  predicated  on  having  analytical  expressions  for  the 
entropy  of  a  given  signal  model,  particularly  in  the  communications  field  (e.g.,  see  [6].  For  some 
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probability  distributions  expressions  for  differential  entropy  are  well  known  (e.g.,  see  Table  17.1  in 
referenee  [2])  while  for  others  sueh  expressions  have  not  yet  been  derived. 

This  work  derives  the  differential  entropy  for  harmonie  signals  with  non-uniform  (Beta)  distributed 
phase.  In  a  number  of  applieations  the  signal  under  study  is  harmonie  and  the  primary  uneertainty 
meehanism  giving  rise  to  the  probability  distribution  is  the  phase  noise  whieh  ean  be  distributed 
non-uniformly.  Optieal  eommunieations  strategies,  for  example,  often  involve  the  phase  modulation  of 
a  earrier  signal  and  subsequent  demodulation  at  the  reeeiver.  The  measured  photoeurrent  at  the  reeeiver 
takes  the  general  form  [7] 

/(f)  =  Ji  -  cos  (A6'(f)  +  A0(f))  j  (1) 

where  A  is  a  sealar  amplitude,  A6*(f)  eontains  the  desired  information,  and  A0(f)  represents  the  noise. 
In  differential  phase  shift  keying,  the  probability  distribution  funetion  assoeiated  with  A0(f)  has  been 
modeled  both  as  a  Gaussian  and  as  a  uni-modal  (but  not  Gaussian)  distribution  [7],  [8].  Speeifioally, 
Ho  [9]  found  the  distribution  to  be  a  eonvolution  of  a  Gaussian  and  a  non-eentral  ehi-squared  distribution 
with  two  degrees-of-freedom. 

An  additional  situation  in  whieh  the  signal  model  (1)  appears  is  in  interferometrie  measurement 
or  deteetion  systems  [10].  In  these  applieations  the  goal  is  still  the  same  as  in  eommunieations:  to 
reeover  the  parameter  A6*  given  phase  noise  A0.  The  work  of  Arie  et  al.  eonsidered  a  Gaussian  phase 
model  [10]  while  Freitas  [11]  eonsidered  a  non-Gaussian  phase-noise  model.  Caleulation  of  the 
differential  entropy  for  the  ease  of  a  sinusoid  with  non-uniform  phase  distributions  does  not  appear 
in  the  literature  however. 

This  paper  will  therefore  derive  the  differential  entropy  for  the  sine  wave  for  the  ease  when  the 
phase  angle  is  Beta-distributed.  The  Beta  distribution  is  an  extremely  flexible  distribution  possessing 
finite  support  and  ean  be  made,  via  appropriate  ehoiee  of  parameters,  to  approximate  a  number  of 
other  well-known  distributions.  The  uniform  distribution,  for  example,  is  a  speeial  ease  of  the  Beta 
distribution.  It  will  be  shown,  in  faet,  that  for  appropriate  ehoiee  of  parameters  the  differential  entropy 
for  the  Beta-distributed  phase  ease  reduees  to  that  of  the  well-known  expression  for  differential  entropy 
assoeiated  with  a  uniform  distributed  phase. 

2.  Mathematical  Development  and  Results 

The  following  notation  will  be  utilized.  The  differential  entropy  for  a  eontinuous  random  variable,  X, 
with  probability  density  funetion  p{x)  is  given  by 

h{X)  =  —  J p{x)  log\p{x)]dx  (2) 

s 

where  S  =  {x\p{x)  >  0}  is  the  support  set  of  X.  Here  log  usually  means  log2.  If  the  log  is  taken  to  the 
base  e,  the  notation  he{X)  is  used.  Consider  the  sinusoidal  signal  written  as: 


Y  =  A  sin  6 
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where  6  is  uniformly  distributed  on  [— 7r,7r]  (A  similar  diseussion  applies  if  F  =  Asin(a;6')  or 
Y  =  Asin(6*  +  0),  where  a;  and  0  are  eonstants).  It  is  well  known  that  this  signal  has  a  probability 
density  funetion  given  by 

-  A  <y<  A  (3) 

and  possesses  zero  mean  and  variance  cr^  =  A^/2  [12].  For  this  distribution  the  associated  differential 
entropy  can  be  computed  by  introducing  the  transformation  z  =  -^  +  \  which  yields  the  probability 
density  function 


p{y)  = 


•n\J  A?  — 


p{z) 


7r^2;(l  -  z) 


r[i] 


r[|]ri 


(4) 


where  r(-)  is  the  Gamma  function.  Equation  (4)  defines  the  Beta  distribution,  with  differential  entropy 
given  by 


K(Z)  = 

=  ln(7r)  —  7  —  2  ln(2)  +  7 


where  ■)  is  the  Beta  function  and  'k(-)  is  the  Digamma  function  (see  Cover  &  Thomas  [2]  p.  486) 
so 

h{Z)  =  log2  (j)  bits.  (6) 

Consequently 


h{Y)  =  h{2AZ  -  A)  =  h{2AZ)  =  h{Z)  +  \og2  {\2A\) 

=  log2  (^)  (7) 

Note  that  h(Y)  can  be  positive  or  negative,  depending  on  the  value  of  A.  This  is  a  well  known  result, 
e.g.,  see  [6].  Indeed  the  entropy  of  arbitrary  Beta  distributions  is  known  [13].  Next  it  will  be  shown 
that  the  result  given  by  Equation  (7)  is  a  special  case  of  the  differential  entropy  of  a  sine-wave  with  a 
Beta-distributed  phase  angle. 

The  probability  density  function  for  a  Beta-distributed  random  variable  on  [0,1]  is  expressed  as: 


fix) 


where  7  >  0,  7  >  0. 

The  transformation  9  =  tv  {x  — 


rjri  +  y)  1 

mm 


{i-xm 


0  <  X  <  1 


provides 


fiO) 


1  r(r7  +  7)  n  ey  Wi  ey  ^ 

7rT{ri)T{'y)  y2  vry  \2  vry  2““2 


(8) 


(9) 
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as  the  Beta  distribution  on  the  interval  f]-  Consequently,  the  sine  wave  Z  =  74sin6*,  where  6  is 
Beta-distributed  on  |]  has  probability  density  funetion: 


p{z)  = 


1  r(??+7)  n  e\7-i  /i  _  e\ 

TT  r(r;)r(7)  \2  '  n  /  \2  tt/ 


A  cos  6* 


ir(r,  +  7)  a  + 


z\V-^ 


7rT{r])T{y) 


^fA 


2  ^2 


A  <  z  <  A 


The  differential  entropy  for  Z  in  nats,  is  ealeulated  as  follows: 


h,(Z)  =  -  - 


-A 


In 


Trr(7])r(j) 


1  r(>?+7)  n  ,  1  j  -1  A  f  i  -  i  sin-^ 

7r  r(r;)r(7)  V  2  ^  TT At  V2  tt  At 


dz 


Letting  te  =  sin  ^  (^),  so  2;  =  A  sin  te  and  dz  =  Acosw  dw,  gives: 


(10) 


(11) 


irh  +  7)  d  +  fr‘(i-f) 


h,{Z)  =  -  /  - 


r 


7rr(r7)r(7) 


Acoste 


-h(7  -  1)  In  (  ^  +  — ^  +  (7  -  1)  In  -  ln(y4cosM;) 

2  TT  1  \  2  TT 


A  cos  wdw 


so 


(12) 


he{Z)  =  \n[nB{r],y)]  - 


7rB{p,y) 

+  (7  -  1)  / 


(7-1) 


1  w 

2  +  7 


1  te 

2  +  7 


7-1 


1  wV  7  /I  w\ 

-2-A 


7-1 


1  w 

2  ~  7 


'7-1 


In  (  ^  -  7  )  die 


1  w 

2  +  7 


7— 1  \  n—1 

/I  y;  \  ' 

2  ~  7 


ln(y4coste)(ite 


(13) 


where  we  have  made  the  substitution  for  the  Beta  funetion  B{p,  7)  = 


Setting  M  =  1  +  -  in  the  first  integral  of  Equation  (13)  leads  to: 


_  r(r?)r(7) 

r(?7+7) 


1 

7  “"‘(1 

0 
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which  is  equal  to 

7iB{r],y)[^{'y) +  r])]  (14) 

from  formula  4.253(1)  of  Gradshteyn  and  Ryzhik  [14].  Similarly,  setting  v  =  \  —  ^  m  the  seeond 
integral  gives: 


7r5(77,7)[4'(r/)  -^{y  +  rj)] 


(15) 


Finally,  setting  u  =  |  +  -  in  the  third  integral  and  noting  that  sin  nu  =  sin(te  +  f )  =  cos  w,  we  obtain 


1  w\^  ^  /I  w 


77-1 


— I - I  I - ]  lrL{Acosw)dw  =  71  /  ^(1  — m)^  ^  ln(74sin7rM)(iM 

2  TT  1  \2  TT  /  J 

0 


=  TT 


1  1 

\nA  J  —  uy~^du  +  J  —  uy~^\n{sm7iu)du 

0  0 


=  7ri?(7, 7)  InA  +  TT  J  V?  ^{1  —  uy  ^  ln(sin7rM)(iK 
0 


(16) 


Colleeting  terms  gives 


he{Z)  =\n[7iAB{r],'y)]  -  (7  -  1)4'(7)  -  (v  -  +  (^  +  7  “  2)4'(7  +  7) 

1 

+  — - r  [  \n(sm7iu)du  (17) 

B{v,V  J 

0 

The  last  term  is  the  average  of  the  funetion  ln(sin  wujdu  over  the  Beta  distribution.  Unfortunately,  there 
does  not  appear  to  be  an  analytie  solution  for  this  integral. 

A  variety  of  Beta  distributions  (taken  from  Hahn  and  Shapiro,  [15])  are  shown  in  Figure  1,  along 
with  the  eorresponding  differential  entropy,  h,  for  a  sine  wave  of  amplitude  A  =  1.  These  entropies  are 
expressed  in  bits,  so  the  values  of  h^,  obtained  from  Equation  (17)  are  divided  by  In  2.  The  last  term 
in  (17)  is  ealeulated  by  standard  numerieal  integration  teehniques. 
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Figure  1.  Beta  distributions  with  different  parameter  values  and  the  assoeiated  entropy 
values,  h,  (in  bits). 


X 


X 


We  now  derive  an  analytie  approximation  for  the  integral  term  in  Equation  (17)  whieh  is  valid  when 
r]  >  1  and  7  >  1.  This  teehnique  is  based  on  the  following  integral  found  in  Gradshteyn  and  Ryzhik 
([14],  formula  3.768(11)): 

1 

j  u'^~\l-u)'^~^smaudu  =  -^B{r],y)[iFi{y;y  +  r];ia)-  y  +  p; -ia)]  (18) 

0 

and  its  eompanion  (formula  3.768(12)): 

1 

J  u^~\l  -  u)^~^  cos  aw  du  =  ^5(7,7)  [iFi(7;7  +  p;ia)  +  iFi(7;7  +  p;  -ia)]  (19) 

0 

both  expressed  in  terms  of  generalized  hypergeometrie  series.  In  faet,  iFi  is  defined  by: 
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(20) 


where  (a)„  denotes  the  produet  a(a  +  l)(a  +  2)  ■  •  •  (a  +  n  —  1)  for  n  >  0  and  (a)o  =  1.  {P)n  is  similarly 
defined.  Note  that  Mathematiea  also  gives  expressions  for  the  integrals  in  (18)  and  (19);  these  formulas 
are  in  terms  of  the  2F3  generalized  hypergeometrie  series  and,  with  a  bit  of  manipulation,  ean  be  shown 
to  be  equivalent  to  the  expressions  above. 

We  make  use  of  the  power  series  expansion: 


whieh  converges  for  |x|  <  1,  and  apply  it  to  a:  =  1  —  sinTrw  since  |1  —  sinTrwl  <  1  for  0  <  m  <  1. 


This  gives 


We  wish  to  choose  N  such  that  the  first  N  terms  of  this  series  converge  closely  enough  to  ln(sin7rM) 


for  purposes  of  calculating  the  integral  in  Equation  (17).  We  have  found  that  this  approximation  will 
be  valid  when  r]  >  1  and  7  >  1.  If  either  7  or  7  is  less  than  unity,  then  the  contribution  to  the  integral 
near  the  endpoints  at  m  =  0  and  u  =  1  will  exceed  the  precision  of  the  power  series  convergence  near 
those  points. 


Accordingly, 


Collecting  the  coefficients  of  like  powers  of  sin  nu,  we  can  write  In  (sin  nu)  as 


N 


(21) 


j=0 


where 


k=l 


and 
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This  is  an  unusual  approximation  since  it  is  clear  that  the  coefficients  diverge  as  cx).  However, 

the  coefficients  have  alternating  signs  and  our  calculations  show  that  the  approximation  is  increasingly 
good  for  small  values  of  N  up  to  about  N  =  50,  at  which  point  we  begin  losing  precision.  For  example 
consider  the  approximation  obtained  with  iV  =  24  terms.  In  this  case  the  coefficients  are  given  by  the 
values  shown  in  Table  1. 


Table  1.  Coefficients  for  the  approximation  of  ln(sin(7rM))  using  iV  =  24  terms. 


j 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

d,{N) 

3.776 

-24 

138 

-674.61 

2656.5 

-8500.8 

22433 

-49443 

91934 

-145278 

196126 

-226922 

225346 

j 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

djW 

192011 

140090 

-87167 

45967 

-20359 

7477.6 

-2237.1 

531.30 

-96.38 

12.55 

-1.043 

0.0417 

We  further  expand  Equation  (21)  by  expressing  powers  of  sinx  as  linear  combinations  of  sines  and 
cosines  of  multiples  of  x  (see  Gradshteyn  and  Ryzhik,  [4]  pp.  25-26).  In  general,  for  any  positive 
integer  n. 


(sinx)  = 


22r. 


n—l 


n—k 


k=0 


2n 


,  cos(2(n  —  k)x)  + 
k  1  \  n 


2n 


(22) 


and 


n—  1 


smx 


,2n-l 


22n-2 


n-\-k—l 


k=0 


2n-l 
k 


sin((2n  —  2k  —  l)x) 


(23) 


Combining  (21),  (22),  and  (23)  leads  to  the  conclusion  that  ln(sin7rw)  is  (approximately)  expressible  as 
a  linear  combination  of  terms  of  the  form  sin(m7rM),  cos(m7rM),  and  constant  terms.  Consequently,  the 
integral  in  Equation  (17): 


u 


7-1 


(1  —  m)^  ^  ln(sin7rM)(iM 


can  be  written  as  a  linear  combination  of  integrals  of  the  form: 


u 


7-1 


{1  —  uY  ^du  =  B{ri,'y) 


^(1  — m)^  ^  sui{m'Ku)du 


1 

—  m)^“^  cos{rmiu)du 

which  are  given  analytically  by  Equations  (18)  and  (19). 

Results  for  this  analytic  approximation  are  shown  in  Table  2  for  two  values  of  N.  The  last  term  of 
Equation  (17)  is  shown,  as  well  as  the  resulting  value  of  differential  entropy  h^^Z)  in  nats.  Again,  the 
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amplitude  of  the  sine  wave  is  taken  to  be  A  =  1.  Clearly  the  approximation  is  a  good  one,  particularly 
for  N  =  45.  We  find  that  for  a  wide  variety  of  Beta  parameters  the  error  in  the  approximation  is  <  5%. 
For  larger  values  of  N,  we  find  that  the  quality  of  the  results  degrade.  Specifically,  we  find  that  the  values 
in  the  range  40  <  iV  <  50  give  the  best  results. 


Table  2.  Accuracy  of  the  Analytical  Approximations  of  the  Entropy  for  Various  Values  of 
?7  >  1  and  7  >  1. 


V  7 

^HsmTTu)du 

Entropy  he{Z)  (in  nats) 

Numerical  Integration 

Analytical  Approximation 

Numerical  Integration 

Analytical  Approximation 

II 

to 

II 

II 

to 

A  =  45 

1 

1 

-0.6931 

-0.6677 

-0.6793 

0.4516 

0.4771 

0.4654 

2 

2 

-0.3278 

-0.3268 

-0.3274 

0.6919 

0.6928 

0.6922 

3 

3 

-0.2141 

-0.2140 

-0.2141 

0.6627 

0.6628 

0.6628 

5 

5 

-0.1264 

-0.1264 

-0.1264 

0.5377 

0.5377 

0.5377 

2 

1 

-0.6931 

-0.6677 

-0.6793 

0.2584 

0.2839 

0.2723 

1.5 

3 

-0.4970 

-0.4916 

-0.4948 

0.3747 

0.3801 

0.3769 

1.5 

5 

-0.7485 

-0.7377 

-0.7443 

-0.1837 

-0.1729 

-0.1794 

3 

1.5 

-0.4970 

-0.4916 

-0.4948 

0.3747 

0.3801 

0.3769 

5 

1.5 

-0.7485 

-0.7377 

-0.7443 

-0.1837 

-0.1729 

-0.1794 

3.  Conclusions 

This  paper  provides  an  analytical  approximation  for  the  differential  entropy  of  a  sine  wave  with  a 
Beta-distributed  phase  angle.  The  results  predicted  by  the  expression  are  in  good  agreement  with  those 
obtained  via  numerical  integration  for  Beta  distribution  parameters  7  >  1  and  7  >  1.  For  all  parameter 
combinations  we  have  looked  at,  the  error  in  the  approximation  is  <  5%  when  a  reasonable  number  of 
terms  are  used  in  the  approximation  {N  =  45).  The  result  for  a  uniformly-distributed  phase  angle  is 
also  included  as  a  special  case  of  our  more  general  result.  The  derived  expression  may  prove  useful  in 
entropy-based  calculations  for  signals  in  which  a  non-uniform  phase  distribution  model  is  appropriate. 
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